TECHNIQUES OF PARALLELIZATION IN MARKOV CHAIN MONTE CARLO METHODS By VIKNESWARAN GOPAL A DISSERTATION PRESENTED TO THE GRADUATE SCHOOL OF THE UNIVERSITY OF FLORIDA IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY UNIVERSITY OF FLORIDA

of Dissertation Presented to the Graduate School of the University of Florida in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy TECHNIQUES OF PARALLELIZATION IN MARKOV CHAIN MONTE CARLO METHODS By Vikneswaran Gopal August 2011 Chair: George Casella Major: Statistics Parallel computing is at the forefront of statistical research today. The main reason for this is the fact that it is the most scalable solution, both in terms of cost and computational ability, to the large applied problems that are being solved by contemporary researchers. As many of these solutions utilize Markov chain Monte Carlo techniques, here we investigate methods of running these chains in parallel. For a geometrically ergodic Markov chain, we begin by establishing that the most theoretically sound method of running it in parallel is through the use of regeneration, which divides the chain into truly independent blocks, or tours. Then we continue by investigating some operational issues relevant to parallelizing the regenerative method. Using renewal theory, we show that it is possible to use the information from the completed tours to complete the unfinished ones. We also derive a lower bound on the possible speed-up that can be attained by using our parallel algorithm, as compared with the sequential one. Finally, we document and provide a general purpose software package for running a minorized geometrically ergodic Markov chain on a cluster. We demonstrate our parallel methodology and the use of our software package on two problems of general interest hierarchical linear mixed models and topic models. Finally, as an aside, we also investigate running adaptive Markov chains using a pipeline algorithm. We show that it has similar convergence properties to the sequential algorithm, but provides a substantial speed-up in clock-time.